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I.  INTRODUCTION 

In  the  recent  years,  a  number  of  authors  have  studied  the  problem 
of  selecting  the  "best"  or  a  "good"  subset  of  regression  variables  in  the 
format  of  selection  and  ranking  theory.  Arvesen  and  McCabe  [1]  considered 
a  procedure  for  selecting  the  best  model  from  among  all  reduced  models 
involving  r  (fixed)  out  of  p  independent  variables.  Huang  and  Panchapakesa 
[4]  discussed  the  problem  of  eliminating  all  inferior  models  using  the 
criterion  of  expected  residual  sum  of  squares  to  define  inferior  models. 

Hsu  and  Huang  [3]  investigated  a  sequential  procedure  for  selecting  good 
regression  models. 

Consider  the  usual  linear  model 

(1.1)  Y  =  X6  +  e 

where  X  =  [1,  X]»«*-»Xp_^]  is  an  nxp  matrix  of  known  constants, 

§  =  (Bq.B^. . . ,Bp_^)  is  a  Ixp  vector  of  unknown  parameters  and 

6  N(0,  og^n^"  ^n  ^be  identity  matrix  of  order  nxn  and 

a  =  (a,a,...,a).  The  model  (1.1)  having  p-1  independent  variables  is 
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considered  as  the  true  model.  Any  reduced  model  whose  "X  matrix"  has  r 

columns  is  obtained  by  retaining  any  r-1  of  the  p-1  independent  variables, 

where  2  <  r  <  p-2.  There  are  =  (^"I)  such  models.  These  reduced 

__r  r  r-i  r 

models  of  "size"  r  are  indexed  arbitrarily  by  i  going  from  1  to  k^.  We 
will  refer  to  a  typical  model  in  this  consider  as  Model  ri ,  A  reduced  model 
of  size  r  can  be  written  as 

(1.2)  Y  =  i  =  l,2,....k^. 

where  X  •  and  are  obtained  from  X  and  e  corresponding  to  the  variables 

2 

tnat  are  retained  in  the  model,  and  N(0, 

It  should  be  first  noted  that  our  comparisons  of  models  are  made 
under  the  true  model  assumptions.  Any  reduced  model  with  the  associated 
error  variance  is  called  inferior  if  ^  aoq  where  a(>  1)  is  a  specified 
constant.  Our  goal  is  to  eliminate  all  inferior  models,  from  the  set  of 
2^"^-l  regression  models  including  the  true  model.  For  this  purpose,  we 
consider  a  family  of  hypotheses  testing  problems,  namely,  against 

i  =  l,...,k^;  r  =  2,...,p.  Rejecting  Hg  will  mean  declaring  Model 

ri  to  be  inferior.  We  derive  our  rule  in  Section  III  subject  to  controlling 
errors  as  explained  therein. 


II.  PRELIMINARIES 


For  any  r,  2  _<  r  _<  p,  we  know  that 

(2.1)  SS^.  = 

=  Y'Qri!.  say. 
and 


under  the  true  model , 


(2.2) 


3 


where  the  degrees  of  freedom  =  n-r»  and  the  noncentrality  parameter 
=  (X3) 'Q^^-(X3)/2,  1  1  i  1  k^.  We  note  that  is  not,  in  general, 
zero  and 


(2.3) 


2  _  2  .  2  2, 

°ri  -  ^  ^  Vri* 


It  is  clear  from  (2.3)  that  x^^  should  not  be  large  for  a  good  model. 
For  convenience,  we  use  3^^-  to  denote  the  vector  obtained  from 


3  =  (®o’®l ” ■ ' ’®p-l^  replacing  with  zeros  the  3j  associated  with  variables 
that  are  dropped  from  the  full  model.  For  examples,  when  p  =  6,  r  =  4 
and  the  variables  that  are  retained  are  X^,  and  X^,  we  have 
3  ”  (3q,Bi,32»  ~  ^^0’  ^1’  ^3* 

ip-j  “  ^1*  ^3’  ^4’  * 


Finally,  we  let 


(2.4) 
and 

(2.5) 


%,ri  *  ^iril^ri  ® 


^1 ,ri  ~  ^^ri l^ri  —  ^r^’ 


where  i  =  1,2,. ..,k^;  r  =  2,...,p,  where  X^.  =  v^(a-1)/2. 


III.  DERIVATION  OF  THE  RULE 

As  we  explained  in  Section  I,  our  rule  is  based  on  a  family  of  hypothesis 

p  kr  p  *^r 

testing  problems.  Let  n,  =  u  U  n,  ,  and  =  n  n 

I  r=2  i=l  ^  r=2  i=l 

Consider  the  following  family  of  hypotheses  testing  problem. 

(3.1)  §„.€iio  « 

1  =  l...-.k^,  r  =  2,...,p.  Let  cpp.j  be  the  test  function  for  Hg  vs 
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Then  the  simultaneous  test  of  all  the  hypotheses  in  (3.1)  is  defined  by  the 
vector  cp  whose  components  arecp^^,  i  =  l,...,k^;  r  =  2,...,p.  The  power 
function  of  the  test  is  vector  of  the  power  functions  P^-j(i^-j)  the 
individual  tests,  where 

P  -(6  •)  =  Es  cp  -(V). 

^n'=rr  S..,-  rv- 
-n 

Let  S(a)  be  the  set  of  all  tests  <p^.,  i  =  1 . k^;  r  =  2,...,p  such 

that 

(3.2)  Eg^_cp^^(Y)  <  a.  §  €  fiQ, 

where  a  is  the  specified  value,  i  =  l,...,k^;  r  =  2,...,p. 


2  2 
Case  1:  When  we  estimate  oq  and  use  it  as  the  known  value  of  oq,  i.e., 

2 

we  treat  cg  as  known. 

Since 


r  ri 


is  distributed  with  noncentral  chi-square  x  (v^,x^^.),  we  denote  the  probability 
density  of  S  •  as  g,  (s  •),  i  =  1 . k  ;  r  =  2 . p.  For  any  r  and  i ,  we 


’X  .  ri 

Ar^  ri 


define 


II,  if  9;^  (s^^.)  >  cgp(s^^.), 

0,  if  <  cgo^Spi). 

such  that  Eg  cp5.(Y)  =  a,  €  Jig,  where  s^^.  is  the  observed  value  of 

and  gg(s^^)  is  the  central  chi-square  probability  density.  It  can  be  shown 
thatcpj^,  i  =  l,...,k^;  r  =  2 . p,  maximize 
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_  (SS^j-SSp^)/(p-r) 


ri  SSp^/(n-p) 

is  distributed  with  noncentral  F  distribution  F(p-r,  n-p;  )  with 
noncentral  parameter  x^. ,  we  denote  the  probability  density  of 
as  f.  and 

Ari 


^ri®  '  2  '  ^  2  '  /P-rJ/  ^ri 

J!r(^)r(!!f)  '"‘P'  'u  ^  v^, 


■o'Vi'  j=0 


is  strictly  increasing  in  ,  for  i  =  k^;  r  =  2,...,p. 


We  define 


Im 


such  that  Es  =  «»  6^,-  €  fln*  where  v^..  is  the  observed  value 

ri  “  -ri  u  ri 

of  and  ^q(v^,-)  is  the  central  F  probability  density.  It  can  also 
be  shown  that  i  =  l,...,k^;  r  =  2,...,p,  maximize 

min  _  inf  Eg  i|x  .(Y) 

r=2,...,p 

among  all  tests  in  S(cx),  i  =  l,...,k^;  r  =  2,...,p. 

The  rule  is  equivalent  to  the  following 


I  if  *Hi'‘rr 


0  If  v^,  <  d^,, 


where  is  determined  by 
n 
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P{V  .  >  <i.\X.  =  0}  =  a. 
n  —  n '  n 

Note  that  the  power  functions  P{S^^-  ^  x^}  and  PtV^^.  2^ 

are  increasing  in  x  •• 

^  ri 

Example: 

We  use  Hald  data  (Draper  and  Smith  (1981))  to  discuss  the  procedure 
as  follows: 


NO 

'1 

'2 

'3 

'4 

Y 

1 

7 

26 

6 

60 

78.5 

2 

1 

29 

15 

52 

74.3 

3 

11 

56 

8 

20 

104.3 

4 

11 

31 

8 

47 

87.6 

5 

7 

52 

6 

33 

95.9 

6 

11 

55 

9 

22 

109.2 

7 

3 

71 

17 

6 

102.7 

8 

1 

31 

22 

44 

72.5 

9 

2 

54 

18 

22 

93.1 

10 

21 

47 

4 

26 

115.9 

11 

1 

40 

23 

34 

83.8 

12 

11 

66 

9 

12 

113.3 

13 

10 

68 

8 

12 

109.4 

as 


The  regression  model  has  been  established  (Draper  and  Smith  (1981)) 


Yi  =  Bq  +  6^X^.  +  62X2^  +  83X3^.  +  X^^.  +  c.. 


e.  'V'  N(0,O/,),  i  =  1,2,. ..,13. 


1 

Let 


p,  =  min  min  inf  Ecpj4(Y)  =  rain  min  P{S^.  jx^.  = 

'  2<r<p  l<i<k^  X_,>X„  ■  2<rj9  lii<k.  ^ 


Zi'-iP  ■_  _-r  r 


and 


Po  =  min  min  inf  Ei()^.(Y)  *  min  rain  P{V  .  id  Jx  .  =  x  }. 
2<r<p  l<i<k^  Xj..>x^  ■  2<r^  l<i<k^  ^ 
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2  1'^  2 

Case  1:  We  use  the  residual  mean  square  s  =  y.  (Y^.-?)  =  5.98  to 

2  2 
estimate  Oq  as  the  known  value  of  Oq.  Testing 


The  procedure  cp  •  is 


1  if 

0  /  \ 
cpri(i')  = 

0  if  5^, 

<  ^i- 

We  list  the 

process  to  reject  inferior  models  as 

follows: 

For  ct  =  0.05 

Order 

Model 

=  13-r 

r  ri 

V  c  • 
r  ri 

Result 

1 

'1 

11 

211.65 

19.675 

reject 

2 

X2 

11 

151.56 

19.675 

reject 

3 

^3 

11 

324.31 

19.675 

reject 

4 

^4 

11 

147.81 

19.675 

reject 

5 

X1X2 

10 

9.68 

18.307 

not  reject 

Continuing 

in  this  fashion, 

the  final  decision  is 

:  Retain 

the  models 

{X^,X2},, 


Note  that  the  order  can  be  randomly  given. 


We  have  obtained  Y  =  6q  +  e-jX^  +  $2X2  +  e  as  the  desired  reduced 
model.  The  following  table  is  shown  the  behavior  of  p-j  when  a  is  changed. 

0.05 


1.6 

2.0 

2. 

■ 

2.8 

0.30 

0.51 

0. 

69 

0.82 

It  shows  that  p^  increases  rapidly  as  A  increasing 


2:  When  unknown. 


ig 


The  procedure  tj)  is 


♦rt<i)  • 


1  if  Vi 


0  if  v^,  < 


list  the 

process 

to  reject 

inferior 

models  as 

follows:  For 

Order 

Model 

''ri 

=  13-4 

‘^ri 

Result 

1 

^1 

67.88 

11 

4.07 

reject 

2 

X2 

47.85 

11 

4.07 

reject 

3 

'3 

107.23 

11 

4.07 

reject 

4 

^4 

46.60 

11 

4.07 

reject 

5 

^1^2 

0.84 

10 

4.46 

not  reject 

Continuing  in  this  fashion,  the  final  decision  is:  Retain  the  models 

.  Note  that  the  order  can  be  randomly  given. 

We  also  obtain  Y  =  6g  +  B-iX-i  +  82^2  ^  desired  model.  Thi 

table  of  the  relation  between  a  and  P2  as  follows: 


a  =  0.05 


1.6 

2.0 

mm 

2.8 

B 

0.37 

0.57 

0.74 

0.84 

Discussion 

From  the  above  example,  we  can  see  that  the  procedure  cp  and  ii^  are 
easy  to  apply  to  obtain  our  desired  models.  We  also  can  guarantee  the 
most  conservative  power  for  any  model  we  selected. 
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Our  rule  is  designed  to  select  all  models  which  are  good  in  the 
sense  of  having  adequate  precision  in  predicting  compared  to  the  true 
model.  When  the  final  selection  consists  of  several  models,  the  experimenter 
may  be  guided  by  practical  considerations  in  choosing  one  of  these.  For 
example,  a  model  with  smallest  number  of  variables  may  be  a  consideration. 

In  practice,  it  may  be  easier  to  obtain  information  on  some  variables  than 
the  others;  thus  a  model  with  variables  easy  to  handle  will  be  preferable. 

Of  course,  it  may  be  preferable  to  build  a  cost  factor  in  the  problem.  In 
the  long  run, one  may  randomly  choose  a  model  for  the  selected  group. 
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